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A  key  element  in  the  success  of  any  project  or  program  is  the  ability  to  communicate 
progress  against  a  baseline  of  cost,  schedule  and  technical  performance  within  and 
outside  the  team.  When  the  expectations  for  communications  are  not  understood 
clearly  and/or  are  misaligned  horizontally  or  vertically  across  the  program,  it  becomes 
very  difficult  for  all  affected  stakeholders  to  answer  the  questions,  "So  where  are  we 
today?  Where  will  we  be  tomorrow?" 

The  communication  of  metrics  can  facilitate  trust,  illustrate  progress,  identify  issues  and  highlight  the  effective¬ 
ness  of  implemented  process  improvements.  To  achieve  these  benefits,  measuring  and  reporting  should  be  at  the 
heart  of  every  project  including  those  based  upon  Agile  approaches.  However,  projects  or  programs  with  Agile 
content  often  require  their  own  set  of  tailored  metrics  and  traditional  assessments  that  may  not  be  usable  for  the 
entire  stakeholder  set.  This  particular  point  is  an  important  planning  consideration  in  the  Department  of  Defense 
(DoD)  environment  where  there  is  significant  hierarchical  reporting  and  numerous  levels  of  multiple  stakeholders, 
all  with  varying  needs  and  expectations  for  performance  data  and  information. 

By  their  very  nature,  Agile  metrics  are  available  to  be  reported  and  analyzed  more  frequently  since  this  approach 
delivers  projects  through  small,  well-vetted  "sprints."  Each  sprint  has  a  goal,  and  the  assessment  of  achieved 
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functionality  always  is  a  conducted  activity  of  the  sprint 
with  the  user  representative. 

Given  the  increased  frequency  and  quantity  of  available 
metrics,  Agile  teams  need  to  highlight  only  the  most 
vital  and  timely  metrics.  What  "vital  and  timely"  means 
to  various  stakeholders  is  where  the  real  crux  of  plan¬ 
ning  resides:  Determine  the  needs  and  expectations  of 
performance  reporting  at  all  stakeholder  levels.  There 
is  a  requirement  to  align  reporting  across  all  levels  of 
the  government-vendor  team.  This  requires  matching 
the  traditional  DoD  project  monitoring  methodology 
(focusing  on  tracking  the  performance  of  each  work 
breakdown  structure  [WBS]  work-package)  to  that  of 
Agile  methodology  (where  tracking  is  focused  on  incre¬ 
mental  delivery  of  functional  capability).  Within  DoD 
acquisition,  we  have  to  structure  our  solicitations  to 
accommodate  these  different  requirements  during  the 
project/program  execution. 

In  planning  performance  reporting,  each  stakeholder 
group  should  receive  only  the  metrics  relevant  to  its 


needs  and  expectations.  Emerging  best  practices  within 
the  software  development  community  have  identified  a 
potential  set  of  criteria  for  establishing  metric  require¬ 
ments  for  various  stakeholder  groups  working  on  Agile 
programs: 

■  Relevance  to  their  decision-making  affecting  the 
project/program 

■  Sufficiency  of  detail  to  be  usable 

■  Availability  (e.g.,  daily,  for  an  iteration  or  release, 
or  a  milestone/gateway  ...  etc.)  for  their  roles  and 
responsibilities 

To  be  effective,  a  proposed  model  for  tailoring  Agile 
metrics  would  be  based  around  commonly  definable 
stakeholder  groups.  Within  DoD,  a  potential  set  of 
groups  could  include  direct  team  members,  senior 
sponsors/leaders,  organizational  stakeholders  and  ex¬ 
ternal  stakeholders/users.  Understanding  the  compo¬ 
sition  of  these  groups,  and  establishing  a  set  of  specific 
expectations  each  would  have  for  performance  metrics 
would  promote  development  of  specific  information 
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requirements  that  can  be  articulated  within  the  body  of  a 
contractual  vehicle. 

Stakeholder  Needs — Team  Members 

More  than  any  other  audience,  team  members  need  highly 
specific  and  detailed  information  because  they  have  the  great¬ 
est  immediate  use  for  such  data. 

This  particular  group  is  involved  in  the  daily  efforts  associated 
with  the  planning,  development,  testing  and  delivery  of  soft¬ 
ware  to  support  functional  capability  requirements.  Therefore, 
this  group  needs  highly  specific  and  detailed  information  that 
is  immediately  available  for  use.  These  data  must  quickly  de¬ 
scribe  what  is  happening  with  the  project  (at  the  sprint  level), 
provide  a  means  to  diagnose  issues,  identify  areas  for  improve¬ 
ment  and  provide  positive  incentives  for  the  team.  The  intent  is 
to  select  only  the  best  metrics  that  give  teams  the  detail  they 
need  without  overwhelming  them. 

Research  on  the  evolving  best  practices  within  the  Agile  de¬ 
velopment  community  indicates  that  a  set  of  performance 
information  focused  on  the  team  member  stakeholders  likely 
would  consist  of  the  following  commonly  available  metrics: 

■  Velocity:  The  number  of  features  a  team  can  deliver  during 
a  sprint  is  the  principal  Agile  metric,  as  it  allows  the  team 
to  accurately  predict  and  plan  progress,  thereby  keeping 
projects  on  schedule  and  within  budget. 

■  Burn  Up/Burn  Down  (BU/BD):  A  burn-up  chart  shows 
how  many  features  the  team  has  promised  to  deliver,  while 
a  burn-down  chart  shows  how  many  features  it  has  com¬ 
pleted.  The  real  power  of  these  charts  to  the  team  mem¬ 
bers  is  motivation.  They  permit  team  members  to  clearly 
see  when  they  are  likely  to  finish  the  project  and,  in  com¬ 
parison,  to  see  the  steady  reduction  of  the  work  still  to  be 


done.  This  particular  metric  enhances  the  team's  ability  to 
answer  earned  value  management  (EVM)  questions  about 
"what  value  has  been  earned  and  what  is  left  to  complete." 
■  Running  Tested  Features  (RTF)/Defect  Density:  For  all 

software  development  projects/programs,  understanding 
defects  has  been  a  standard  metric  and  is  a  completely  ap¬ 
plicable  and  critical  quality  metric  in  the  Agile  environment. 
RTF,  a  similar  measurement,  shows  how  many  features  in 
each  sprint  have  passed  acceptance  tests.  As  with  the  BU/ 
BD  metric,  positive  data  can  be  very  motivating  to  the  de¬ 
velopment  team.  In  practice,  Agile  techniques  such  as  "test- 
driven  development"  and  "acceptance  test-driven  develop¬ 
ment"  contribute  significantly  to  the  prevention  of  defects. 
Not  introducing  defects  into  the  system  in  the  first  place  will 
greatly  reduce  "defect  density"  when  compared  with  the 
more  traditional  DoD  software  development  approaches. 

Stakeholder  Needs — Senior  Sponsors 
and  Leadership 

For  senior-level  leadership  of  both  Agile  and  non-Agile  proj¬ 
ects,  traditional  metrics  are  still  the  most  appealing.  For  these 
stakeholders,  the  strategic  concerns  of  the  project  or  program 
are  chief  concerns.  For  this  group,  the  primary  focus  is  under¬ 
standing  whether  the  project  is  on  budget  and  schedule  and 
going  to  deliver  the  promised  performance.  As  a  general  rule, 
the  details  of  issues  such  as  defects,  unless  they  affect  the 
cost,  schedule  or  capability  of  the  software,  are  not  important. 

At  this  touchpoint  in  the  DoD  hierarchy  of  senior  leaders  and 
project  team  members,  the  real  difficulty  in  translating  metrics 
occurs.  Agile  metrics  differ  from  traditional  metrics  in  that 
they  are  considered  "adaptive"  rather  than  "predictive."  In  a 
traditional  waterfall  project,  the  cost,  time  and  desired  capabil¬ 
ity  are  defined  at  initiation;  therefore,  the  metrics  emphasize 
planned  values  (the  Budget  Cost  of  Work  Scheduled  [BCWS] 


Figure  1.  Sample  Burn  Down  Chart 
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from  an  EVM  perspective).  In  an  Agile  development  project, 
these  constraints  (BCWS)  will  evolve  as  a  function  of  the  qual¬ 
ity  of  the  software  completed;  the  emphasis  shifts  to  metrics 
focused  on  earning  value  (Budgeted  Cost  of  Work  Performed 
or  BCWP). 

In  a  fashion  similar  to  the  discussion  of  team  members,  re¬ 
search  on  the  evolving  best  practices  within  the  Agile  de¬ 
velopment  community  indicates  that  a  set  of  performance 
information,  focused  on  the  senior  sponsors  and  leadership 
stakeholders  likely  would  be  built  around  the  following  metrics: 


committed  to,  what  it  has  accomplished  so  far  and  what  it 
has  yet  to  deliver. 

Potentially,  there  are  other  persons  affiliated  with  the  senior 
sponsor  and  leadership  groups  who  have  an  interest  in  a  proj¬ 
ect  but  aren't  working  directly  on  it.  This  group  would  include 
roles  such  as  the  program  manager(s), Fleet  liaison,  resources 
and  other  functional  managers.  They  generally  will  be  inter¬ 
ested  in  the  same  high-level  business  metrics  as  the  senior 
sponsors  and  leadership,  though  they  often  require  additional 
details  related  to  their  specific  functions. 


■  Bum  Down  (BD):  The  senior  sponsor  and  leader  version  of 
a  burn-down  report  would  summarize  from  a  high  level  how 
many  required  performance  capabilities  or  features  have 
been  delivered  and  how  many  remain  outstanding.  To  facili¬ 
tate  this  reporting,  there  needs  to  be  a  clear,  traceable  and 
unambiguous  systems  engineering  discipline  of  "threading” 
the  user-based  requirements  (Key  Performance  Parameters 
and  Key  System  Attributes)  and  the  buyer-based  require¬ 
ments  (Specifications,  Statement  of  Objectives  and/or 
Statement  of  Work-related)  down  to  the  capabilities  being 
provided  with  each  Agile  sprint. 

■  Earned  Business  Value  (EBV):  EBV  is  a  commercial  sec¬ 
tor  practice  that  communicates  an  Agile  project's  progress 
toward  delivering  its  expected  goals.  It  may  be  adaptable  to 
the  DoD  environment  since  it  is  related  to  similar  principles 
that  allow  for  the  use  of  an  EVM  system.  In  practice,  when 
items  from  the  product  backlog  (the  remaining  agreed-to 
project  performance  capability  yet  realized)  are  completed, 
they  add  to  the  project's  EBV  as  a  percentage  of  its  cumula¬ 
tive  Return  on  Investment  (ROI).  This  percentage  is  deter¬ 
mined  for  each  specific  capability  delivered  during  a  particu¬ 
lar  sprint.  Since  quality  of  the  developed  software  is  an  Agile 
project's  principal  objective,  EBV  as  a  metric  provides  senior 
sponsors  and  leadership  a  measure 
of  how  much  value  has  been  deliv¬ 
ered  thus  far  for  the  end  user.  As  with 
the  "Burn-Down”  above,  strong  link¬ 
age  between  the  individual  "scope” 
of  each  sprint  and  the  high-level  per¬ 
formance  of  the  system  at  the  "user 
perspective”  is  critical  to  the  value  of 
the  EBV  metric. 


A  metric  such  as  EBV  may  prove  too 
complicated  to  articulate  in  data  deliv¬ 
erable  in  your  solicitation  or  to  utilize 
within  the  DoD  program  environment, 
so  an  internal  manipulation  of  perfor¬ 
mance  data  may  be  required  to  meet 
the  expectations  of  senior  sponsors  and 
leadership. 


For  example,  the  Fleet  liaison  team  lead  may  need  to  know 
when  a  software  increment  or  full  capability  will  be  available 
so  the  team  can  plan  and  resource  the  Fleet  implementation 
with  support  engineers  and  the  receiving  activity.  The  metric 
of  "Velocity”  would  not  be  a  useful  metric  on  its  own  for  the 
liaison  team  but  would  become  very  relevant  when  accom¬ 
panied  by  a  direct  detailed  narrative  discussion  on  the  team's 
progress. 

Facilitating  the  exchange  of  information  such  as  this  will  be  a 
key  role  of  the  "Agile  Advocate”  and  "end-user  representative.” 
These  two  roles,  as  discussed  in  the  January-February  2013 
Defense  AT&L  magazine  article  "The  Challenges  of  Being  Agile 
in  DoD,”  form  the  key  bonds  between  the  development  team 
and  outside  world  of  the  program  office  and  other  stakeholders. 

Stakeholder  Needs — External  Stakeholders 

The  external  stakeholders  are  the  group  that  receives  the 
greatest  benefit  from  Agile  approaches  due  to  its  improved 
"time  to  market.”  In  DoD,  this  group  of  stakeholders  includes 
both  the  end-user  in  the  Fleet,  as  well  as  elements  within  the 
parent  service  and/or  at  the  level  of  the  Office  of  the  Secretary 


Figure  2.  Earned  Business  Value  Example 


The  current  DoD  practice  utilizes 
a  "dashboard”  that  fundamen¬ 
tally  can  display  what  the  team  has 
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Agile  metrics  differ  from 
traditional  metrics  in  that  they 
are  considered  "adaptive" 
rather  than  "predictive." 

of  Defense.  If  these  stakeholders  are  funding  the  project,  they 
should  receive  the  same  high-level  business  metrics,  such  as 
EBV,  that  senior  sponsors  and  leadership  get.  Otherwise,  as 
in  the  case  of  the  Fleet  user,  the  metric  they  will  care  about 
most  will  be  whatever  portion  they  get  and  when  their  "vetted 
capabilities  need"  will  be  delivered. 

The  most  compelling  aspect  of  Agile  is  its  iterative  process. 
Software  capabilities  can  start  showing  up  earlier  to  the  Fleet 
user  than  in  the  traditional  process.  Close  coordination  and 
sound  configuration  management  discipline  are  necessary  to 
ensure  that  all  the  needed  elements  are  in  place  for  the  user 
to  accept  these  incremental  capability  enhancements— a  clear 
driver  for  the  proper  set  of  metrics. 

Take  Time  and  Be  Selective 

A  large  array  of  Agile  metrics  is  available  to  project  managers 
and  the  stakeholders  they  team  with.  Because  of  the  nature 
of  Agile  (i.e.,  an  emphasis  on  speed),  it  demands  that  project 
managers  choose  their  information  tools  wisely  to  effectively 
integrate  with  the  demands  of  the  team,  sponsors,  leadership 
and  external  customers. 

Aligned  to  Agile  principles,  the  project  team  should  look  to 
measure  the  minimum  necessary  to  satisfy  all  the  stake¬ 
holder  requirements.  DoD  therefore  must  stipulate  what 
performance  reporting  it  desires  at  all  levels  and  allow  the 
development  team  to  propose  ways  to  meet  that  reporting 
requirement.  In  essence,  DoD  needs  to  consider  how  to  focus 
on  providing  a  "statement  of  objective  for  metrics"  to  facilitate 
better  performance  reporting. 

Other  Best  Practices 

The  suggested  metrics  proposed  in  this  article  offer  a  poten¬ 
tial  foundation  for  discussing  what  information  to  present  to 
stakeholders  at  various  levels.  If,  however,  the  stakeholders  on 
your  particular  program  are  not  satisfied  with  your  planned 
approach  to  reporting  performance,  best  practices  suggest 
the  following  strategies  be  considered  to  obtain  buy-in: 

■  Solicit  Examples.  If  your  stakeholders  desire  more  or  differ¬ 
ent  metrics,  ask  them  to  provide  a  template  or  report  format 
consistent  with  their  needs.  This  practice  is  better  served 
prior  to  the  award  of  a  contract  for  development,  when  po¬ 
tential  vendors  can  adjust  their  scope  and  cost  estimates. 


■  Promote  Open  Communication.  Agile  is  fast-paced,  so  offer 
greater  visibility  of  the  information  being  collected.  This  can 
satisfy  those  who  wish  to  analyze  the  development  from 
an  independent  perspective.  This  practice,  however,  can 
create  a  huge  additional  burden  on  those  directly  involved 
in  the  development  process:  having  to  explain  terminology 
and  the  purpose  of  details  well  beyond  the  needs  of  those 
external  to  the  team.  Again,  this  is  an  excellent  opportunity 
for  the  "Agile  Advocate"  to  mediate  between  the  various 
stakeholder  elements. 

■  Encourage  Collaboration.  A  key  stakeholder  seeking 
greater  levels  of  information  actually  may  be  looking  for 
greater  levels  of  involvement.  An  approach  espoused  in 
the  commercial  sector  is  to  make  this  key  stakeholder  a 
"co-owner"  of  the  team's  product  backlog  (the  remaining 
agreed-to  performance  capability  of  the  project  that  is  yet 
to  be  realized)  along  with  the  product  owner.  This  action 
would  ensure  the  stakeholder's  involvement  in  the  "con¬ 
struction  and  grooming"  of  the  product  backlog  continu¬ 
ously  from  initiation  to  closeout  of  the  project. 

Conclusions  and  Summary 

Agile,  while  different  in  approach  than  traditional  software¬ 
intensive  projects  and  programs,  still  has  as  a  central  element 
the  need  for  high-quality  communication  of  cost,  schedule  and 
technical  performance.  The  development  team  seeks  to  instill 
a  sense  of  trust,  illustrate  its  progress  and  facilitate  the  reso¬ 
lution  of  issues  that  affect  all  stakeholders,  team  members, 
senior  sponsors  and  leadership  as  well  as  those  outside  the 
organization. 

To  achieve  these  goals,  the  need  for  metrics  that  are  effective 
measures  across  all  stakeholder  levels  must  be  accommodated 
in  the  program's  acquisition  strategy.  Determining  what  vital 
and  timely  mean  at  all  levels  is  an  early  planning  requirement 
if  stakeholder  expectations  of  performance  reporting  are  to 
be  met.  This  task  requires  cross-matching  the  traditional  DoD 
WBS-based  project  monitoring  methodology  to  that  of  an  Agile 
incremental  functional  capability  monitoring  methodology.  The 
desired  outputs  must  focus  on  supporting  decision  making  by 
delivering  sufficient  and  relevant  details  in  a  timely  fashion  to 
leadership  at  all  levels  of  the  organization— a  desired  accom¬ 
plishment  in  any  program,  let  alone  an  Agile  one. 

Accomplishing  the  above  is  not  always  a  simple  and  straight¬ 
forward  process.  Obtaining  the  proper  level  of  buy-in  on  what 
the  various  stakeholders  believe  is  a  robust  set  of  performance 
metrics  for  an  Agile-intensive  project  or  program  may  necessi¬ 
tate  the  use  of  additional  best  practices.  Strategies  should  look 
to  include  open  solicitations  of  example  templates  or  formats 
(better  served  prior  to  contract  award),  upfront  promotion  of 
open  communications  that  include  relying  upon  your  "Agile 
Advocate"  and  establishing  an  environment  of  collaboration 
through  co-ownership  of  key  program  planning  throughout 
the  development  life  cycle. 


The  author  can  be  contacted  at  william. broadus@dau. mil. 
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